2023-08-02T02:56:29.565 INFO:root:teuthology version: 0.0.1.dev174+g10753fd 2023-08-02T02:56:29.566 INFO:teuthology.results:subset: '204/1024' 2023-08-02T02:56:29.566 INFO:teuthology.results:seed: '7214' 2023-08-02T03:38:11.201 INFO:root:teuthology version: 0.0.1.dev174+g10753fd 2023-08-02T03:38:11.210 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-02T04:35:15.381 INFO:teuthology.results:Tests finished! gathering results... 2023-08-02T04:35:16.993 INFO:scrape:Found 73 jobs 2023-08-02T04:35:17.247 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358056/teuthology.log 2023-08-02T04:35:18.346 INFO:scrape:Found 6 distinct failure reasons 2023-08-02T04:35:18.347 INFO:scrape:Failure: Command failed on smithi053 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' 2023-08-02T04:35:18.347 INFO:scrape:18 jobs: ['7358105', '7358112', '7358090', '7358098', '7358076', '7358122', '7358087', '7358069', '7358062', '7358116', '7358071', '7358100', '7358124', '7358064', '7358110', '7358114', '7358061', '7358077'] 2023-08-02T04:35:18.348 INFO:scrape:suites intersection: [] 2023-08-02T04:35:18.348 INFO:scrape:suites union: ['1-start', '2-services/basic', '2-services/rgw', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04}', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'task/test_extra_daemon_features}', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'test_iscsi_container}}', 'workloads/cephadm_iscsi}'] 2023-08-02T04:35:18.348 INFO:scrape: 2023-08-02T04:35:18.349 INFO:scrape:Failure: Command failed on smithi055 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" 2023-08-02T04:35:18.349 INFO:scrape:36 jobs 2023-08-02T04:35:18.349 INFO:scrape:suites intersection: ['0-nvme-loop'] 2023-08-02T04:35:18.350 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/rgw', '2-services/rgw-ingress', '3-final}', 'agent/off', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'start}'] 2023-08-02T04:35:18.350 INFO:scrape: 2023-08-02T04:35:18.350 INFO:scrape:Failure: list index out of range 2023-08-02T04:35:18.351 INFO:scrape:1 jobs: ['7358074'] 2023-08-02T04:35:18.351 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cluster/{1-node}', 'objectstore/bluestore-bitmap', 'orch:cephadm/nfs/{begin/{0-install', 'overrides/ignorelist_health', 'supported-random-distros$/{rhel_8}', 'tasks/nfs}'] 2023-08-02T04:35:18.351 INFO:scrape: 2023-08-02T04:35:18.352 INFO:scrape:Dead: None 2023-08-02T04:35:18.352 INFO:scrape:1 jobs: ['7358056'] 2023-08-02T04:35:18.352 INFO:scrape: 2023-08-02T04:35:18.352 INFO:scrape:Failure: Command failed on smithi160 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout ddf6e4599db46993f6615743ddf10e187c831512' 2023-08-02T04:35:18.353 INFO:scrape:1 jobs: ['7358120'] 2023-08-02T04:35:18.353 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-08-02T04:35:18.353 INFO:scrape: 2023-08-02T04:35:18.354 INFO:scrape:Failure: Command failed on smithi193 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 177eb68e-30e6-11ee-9b37-001a4aab830c -- ceph mon dump -f json' 2023-08-02T04:35:18.354 INFO:scrape:1 jobs: ['7358084'] 2023-08-02T04:35:18.354 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-08-02T04:35:18.354 INFO:scrape: 2023-08-02T04:35:18.357 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/ failed: 57 dead: 1 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [7358055] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:53 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358055/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358055/ sentry: https://sentry.ceph.com/organizations/ceph/?query=391b8890409c4f199b8f3228ef0ee5b3 Command failed on smithi044 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358057] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:08:11 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358057/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358057/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5be14fd8561144cfa87bb4ace8ec12db Command failed on smithi130 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358058] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:12:31 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358058/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a30850e44e24d4488b86e1e8d5c8911 Command failed on smithi146 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358059] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:10 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358059/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=824521511df7459b800736580583e918 Command failed on smithi090 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358060] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:08:01 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358060/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ae9f9ddde074f2cb8ba08589ccaf0d0 Command failed on smithi117 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358061] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:26:05 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358061/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358061/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aba62169c701402db2b78ff001f1dc4b Command failed on smithi043 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358062] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:10:35 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358062/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358062/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e58acb43c324a88a81c433af95e646c Command failed on smithi038 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358063] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:11:03 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358063/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358063/ sentry: https://sentry.ceph.com/organizations/ceph/?query=221faaa3f50f4f2c91b937a3397ca06f Command failed on smithi019 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358064] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:17:45 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358064/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358064/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab92150725bd42b1b3e5ce0f31bdf388 Command failed on smithi062 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358065] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358065/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358065/ sentry: https://sentry.ceph.com/organizations/ceph/?query=edbb028060524cb2b73fcc3d20847473 Command failed on smithi026 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358067] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:10:43 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358067/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358067/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bd0b88dec6f4b38add0551166784abc Command failed on smithi002 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358068] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:10:59 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358068/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358068/ sentry: https://sentry.ceph.com/organizations/ceph/?query=312e4a7e81fa4f38a61f73c356e318f7 Command failed on smithi123 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358069] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:20:47 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358069/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358069/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc286c5b609c44508754df8c64d25f23 Command failed on smithi018 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358070] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:14:19 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358070/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358070/ sentry: https://sentry.ceph.com/organizations/ceph/?query=924977b24b6145999d02fa1e8b4794e2 Command failed on smithi003 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358071] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:19:24 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358071/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358071/ sentry: https://sentry.ceph.com/organizations/ceph/?query=990a24de09354c7fae29246b32eef825 Command failed on smithi097 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358072] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:13:06 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358072/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358072/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0bfa5efd63846a895b79fa628b9e20f Command failed on smithi023 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358074] orch:cephadm/nfs/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/nfs} ----------------------------------------------------------------- time: 00:16:32 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358074/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358074/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ceb2491c80ee446f9852c258683ed4db list index out of range [7358076] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:11:17 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358076/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358076/ sentry: https://sentry.ceph.com/organizations/ceph/?query=924e20c509714b518067de162459905c Command failed on smithi037 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358077] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:07:52 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358077/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358077/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43a5547a6d11432f8775f3722349c972 Command failed on smithi139 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358078] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358078/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358078/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdb0db9585dd4248a16beeb64924ab18 Command failed on smithi006 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358079] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:27 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358079/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358079/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0a1a6d791ee44bc99add0c25b29010b Command failed on smithi098 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358082] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:11:06 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358082/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358082/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45feae31a94e412ab6f520a8ba06b838 Command failed on smithi032 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358083] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:07:56 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358083/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358083/ sentry: https://sentry.ceph.com/organizations/ceph/?query=123a2531a0464e02ba1ccfe90d7f741a Command failed on smithi045 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358084] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:14 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358084/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d48b153f290642f2ab4afb903609fd74 Command failed on smithi193 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 177eb68e-30e6-11ee-9b37-001a4aab830c -- ceph mon dump -f json' [7358085] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:10:37 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358085/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358085/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96ad82c303f14e5d810f4711d081e921 Command failed on smithi040 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358087] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:14:52 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358087/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358087/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e304133924d742b598de03778bb8156b Command failed on smithi017 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358088] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:10:40 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358088/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa974f9a202d467c9bac9d9b74d2f965 Command failed on smithi170 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358089] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:13:25 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358089/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42c645ddf0ec45f88b3d1bb0114433a3 Command failed on smithi029 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358090] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:17:07 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358090/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=009163c57d87404782b40ea3898be687 Command failed on smithi061 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358092] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358092/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358092/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21d6751ed558491691adca719d233445 Command failed on smithi055 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358093] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:10:54 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358093/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aecadce68d9d4404a1f60833d9047648 Command failed on smithi138 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358095] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:04 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358095/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358095/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e36f56b2046149e3aff76b778e3e8859 Command failed on smithi116 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358096] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:15:51 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358096/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358096/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b97f66b15484e3ca588b3ee9b825217 Command failed on smithi046 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358097] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:35 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358097/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1056a778df8b4620bbd4401053189101 Command failed on smithi162 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358098] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:25:21 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358098/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358098/ sentry: https://sentry.ceph.com/organizations/ceph/?query=231fc85a2dfb40ad988957adb6309567 Command failed on smithi084 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358099] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:13:01 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358099/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358099/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a09d60b46314144ade1b2e3882510e4 Command failed on smithi161 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358100] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:22:51 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358100/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ff3bee9457e4fd28383612ffe5fe7fc Command failed on smithi088 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358103] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:14:31 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358103/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e69b339544484a8d98ba806fc3de6eea Command failed on smithi005 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358104] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358104/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358104/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5df5e895253841ed964d498d69aa2d29 Command failed on smithi087 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358105] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:19:15 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358105/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4bccf32760748d19c8b1e33c5d433e8 Command failed on smithi053 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358106] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:10:47 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358106/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8d44f901d6747c5b2f5e8f8e32ffbcd Command failed on smithi045 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358108] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:08:11 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358108/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358108/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40ee1a9a895d4c4bb21cdef345550f19 Command failed on smithi002 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358109] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:10:27 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358109/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358109/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c96bff38bc5c40879912c5615c086bd9 Command failed on smithi098 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358110] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:20:35 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358110/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358110/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81f6f4b082c842329134c5a62bbfb57a Command failed on smithi146 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358112] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:16:51 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358112/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358112/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fd5310d78544d96a6f230835abe87c9 Command failed on smithi050 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358114] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358114/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358114/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0eefc30bd5924679affe0874f0a7bcf2 Command failed on smithi092 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358115] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:36 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358115/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358115/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6db615ddc181425181e4f88a412dbbf2 Command failed on smithi040 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358116] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:10:29 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358116/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358116/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4f1be389b534c61bf1fb5eed6caec00 Command failed on smithi062 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358117] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:57 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358117/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358117/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7915ab183294ad8a4d92c1773336d05 Command failed on smithi077 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358118] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:14:30 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358118/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358118/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1259cd770e3f46ca84c10a163b3c5cd8 Command failed on smithi029 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358119] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:13:32 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358119/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358119/ sentry: https://sentry.ceph.com/organizations/ceph/?query=765650c194134b698fe869ea5fa1d819 Command failed on smithi130 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358120] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:50 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358120/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358120/ sentry: https://sentry.ceph.com/organizations/ceph/?query=898f73c4ee7b49c29d0b90ca5c0f3b6a Command failed on smithi160 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout ddf6e4599db46993f6615743ddf10e187c831512' [7358121] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:33 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358121/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bc9380943604210a5123f0505d9dee0 Command failed on smithi038 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358122] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:16:54 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358122/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358122/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13679fedc64241d4a8bda6ae69016ca0 Command failed on smithi129 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358123] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:07:52 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358123/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358123/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33435b2075e8430fbe3833318a38f117 Command failed on smithi070 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7358124] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:10:37 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358124/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49cc52416f7a42b5b6cf13b70a939e84 Command failed on smithi138 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [7358127] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:07:58 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358127/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358127/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83b94ff868f8489baaf06e30b868e66e Command failed on smithi026 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" Dead ================================================================= [7358056] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358056/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358056/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7358066] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:41:48 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358066/ [7358073] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:45:20 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358073/ [7358075] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli} time: 00:34:41 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358075/ [7358080] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_adoption} time: 00:13:04 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358080/ [7358081] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:37:45 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358081/ [7358086] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm} time: 00:25:52 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358086/ [7358091] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:50:11 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358091/ [7358094] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos} time: 00:12:13 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358094/ [7358101] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:45:01 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358101/ [7358102] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:41:10 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358102/ [7358107] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:52 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358107/ [7358111] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:41:15 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358111/ [7358113] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/on orchestrator_cli} time: 00:29:40 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358113/ [7358125] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:34:24 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358125/ [7358126] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:45:03 info: http://pulpito.ceph.com/vshankar-2023-08-02_02:56:22-orch:cephadm-main-testing-default-smithi/7358126/