2023-08-30T07:26:50.836 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-30T07:26:50.837 INFO:teuthology.results:seed: '9971' 2023-08-30T07:32:25.275 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-30T07:32:25.283 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-30T08:29:29.863 INFO:teuthology.results:Tests finished! gathering results... 2023-08-30T08:29:30.623 INFO:scrape:Found 16 jobs 2023-08-30T08:29:32.836 INFO:scrape:Found 4 distinct failure reasons 2023-08-30T08:29:32.837 INFO:scrape:Failure: Command failed on smithi045 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 353f735a-4709-11ee-9ab2-7b867c8bd7da -- bash -c 'ceph orch upgrade ls'" 2023-08-30T08:29:32.838 INFO:scrape:4 jobs: ['7382463', '7382462', '7382459', '7382460'] 2023-08-30T08:29:32.838 INFO:scrape:suites intersection: ['4-wait', '5-upgrade-ls'] 2023-08-30T08:29:32.838 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-08-30T08:29:32.839 INFO:scrape: 2023-08-30T08:29:32.839 INFO:scrape:Failure: '2023-08-30T08:13:50.444577+00:00 smithi146 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ' in syslog 2023-08-30T08:29:32.840 INFO:scrape:1 jobs: ['7382472'] 2023-08-30T08:29:32.840 INFO:scrape:suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-08-30T08:29:32.840 INFO:scrape: 2023-08-30T08:29:32.841 INFO:scrape:Failure: Command failed on smithi038 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.38:/foouser /mnt/foo'" 2023-08-30T08:29:32.841 INFO:scrape:4 jobs: ['7382465', '7382469', '7382470', '7382464'] 2023-08-30T08:29:32.842 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-08-30T08:29:32.842 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-08-30T08:29:32.843 INFO:scrape: 2023-08-30T08:29:32.843 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-08-30T08:29:32.843 INFO:scrape:1 jobs: ['7382466'] 2023-08-30T08:29:32.844 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-08-30T08:29:32.844 INFO:scrape: 2023-08-30T08:29:32.845 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/ failed: 10 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7382459] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:35:40 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382459/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6242b189fd0c415fbcdefb8d1c6ac1c1 Command failed on smithi040 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd42532a-4708-11ee-9ab2-7b867c8bd7da -- bash -c 'ceph orch upgrade ls'" [7382460] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:45:02 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382460/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a13006d7781f472fb5e45a7767b9f612 Command failed on smithi039 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4e12fc44-4709-11ee-9ab2-7b867c8bd7da -- bash -c 'ceph orch upgrade ls'" [7382462] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:35:43 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382462/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3405437486384cc0b240e8bc1a14aefb Command failed on smithi088 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fb930c20-4708-11ee-9ab2-7b867c8bd7da -- bash -c 'ceph orch upgrade ls'" [7382463] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:28:06 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382463/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=24bb4824b2484b52bbd96be36f8ab63b Command failed on smithi045 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 353f735a-4709-11ee-9ab2-7b867c8bd7da -- bash -c 'ceph orch upgrade ls'" [7382464] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:23:06 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382464/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08d684ac08044352b59b160be9cc2538 Command failed on smithi081 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.81:/foobucket /mnt/foo'" [7382465] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:21:41 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382465/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b84cedbd12944ec1877ca8650d42ad6e Command failed on smithi038 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.38:/foouser /mnt/foo'" [7382466] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:30:12 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382466/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bdf4120204d48c39c380115ba77beb1 reached maximum tries (301) after waiting for 300 seconds [7382469] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:21:56 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382469/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=761ebf3ba0f14220acb9add2849c6fd9 Command failed on smithi092 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.92:/foobucket /mnt/foo'" [7382470] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:23:11 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382470/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25a183e07a0e4ebb86c22c78f9fc37d4 Command failed on smithi037 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.37:/foouser /mnt/foo'" [7382472] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:40:29 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382472/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382472/ '2023-08-30T08:13:50.444577+00:00 smithi146 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ' in syslog Pass ================================================================= [7382457] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:40:08 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382457/ [7382458] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:26:57 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382458/ [7382461] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:55:34 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382461/ [7382467] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:16:06 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382467/ [7382468] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:21:02 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382468/ [7382471] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:20:36 info: http://pulpito.ceph.com/adking-2023-08-29_11:42:05-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382471/