2023-08-11T11:21:58.239 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-11T11:21:58.240 INFO:teuthology.results:subset: '1/8' 2023-08-11T11:21:58.240 INFO:teuthology.results:seed: '2550' 2023-08-11T11:31:36.120 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-11T11:31:36.124 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-11T12:29:40.677 INFO:teuthology.results:Tests finished! gathering results... 2023-08-11T12:29:41.645 INFO:scrape:Found 15 jobs 2023-08-11T12:29:46.198 INFO:scrape:Found 4 distinct failure reasons 2023-08-11T12:29:46.199 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-08-11T12:29:46.199 INFO:scrape:5 jobs: ['7367199', '7367193', '7367204', '7367191', '7367202'] 2023-08-11T12:29:46.199 INFO:scrape:suites intersection: ['1-start'] 2023-08-11T12:29:46.200 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/16.2.0', '1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-start', '2-nfs', '2-services/jaeger', '3-final}', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-08-11T12:29:46.200 INFO:scrape: 2023-08-11T12:29:46.200 INFO:scrape:Failure: Command failed on smithi046 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9ee3c4c-383b-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-08-11T12:29:46.201 INFO:scrape:4 jobs: ['7367194', '7367203', '7367201', '7367196'] 2023-08-11T12:29:46.201 INFO:scrape:suites intersection: ['4-wait', '5-upgrade-ls'] 2023-08-11T12:29:46.201 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-08-11T12:29:46.202 INFO:scrape: 2023-08-11T12:29:46.202 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2023-08-11T12:29:46.202 INFO:scrape:3 jobs: ['7367192', '7367200', '7367205'] 2023-08-11T12:29:46.202 INFO:scrape:suites intersection: ['task/test_nfs}'] 2023-08-11T12:29:46.203 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_nfs}'] 2023-08-11T12:29:46.203 INFO:scrape: 2023-08-11T12:29:46.203 INFO:scrape:Failure: Command failed on smithi050 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.50:/fake /mnt/foo -o port=2999'" 2023-08-11T12:29:46.204 INFO:scrape:3 jobs: ['7367197', '7367195', '7367198'] 2023-08-11T12:29:46.204 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-08-11T12:29:46.204 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-08-11T12:29:46.204 INFO:scrape: 2023-08-11T12:29:46.206 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/ failed: 15 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7367191] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:38:40 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367191/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c3d3c8fd30742de89769e22739a784c reached maximum tries (301) after waiting for 300 seconds [7367192] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:34:09 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367192/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2453fafea2e400898ed3f56f57e38ee Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) [7367193] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:49:54 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367193/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe0976b831224cfb8092c4bdb3c31fc3 reached maximum tries (301) after waiting for 300 seconds [7367194] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:45:07 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367194/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac1cb89fa932467fa26b2ecb1730716e Command failed on smithi046 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9ee3c4c-383b-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367195] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:17:56 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367195/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ffb4781045e4e258e7ca01152c30808 Command failed on smithi172 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.172:/fake /mnt/foo'" [7367196] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:35:42 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367196/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f11e2ab55ef54206bec4f49bddd798e3 Command failed on smithi027 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f28ecd7a-383b-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367197] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:23:00 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367197/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cbb6479ed214a43b99003cfaaa28d15 Command failed on smithi050 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.50:/fake /mnt/foo -o port=2999'" [7367198] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:20:05 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367198/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367198/ sentry: https://sentry.ceph.com/organizations/ceph/?query=901bc5b3877048ba9329865ab3e7748f Command failed on smithi017 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.17:/fake /mnt/foo'" [7367199] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:49:31 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367199/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=849d128bb94343669224a7920ce76390 reached maximum tries (301) after waiting for 300 seconds [7367200] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:25:02 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367200/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a57b219706144b009a69dc962957b69a Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) [7367201] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:44:44 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367201/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367201/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6b962769c7b463cb56f00cb94dd46d2 Command failed on smithi049 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 50566e40-383c-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367202] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:50:23 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367202/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f228b87ce3264485818c206678ee2d9e reached maximum tries (301) after waiting for 300 seconds [7367203] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:34:46 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367203/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1493229ca3e14611a44405f353d2a896 Command failed on smithi039 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7dddc32-383c-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367204] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:27:52 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367204/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04bd9fe9f56f40ed8e5371b9eaaa778f reached maximum tries (301) after waiting for 300 seconds [7367205] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:29:43 info: http://pulpito.ceph.com/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367205/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_11:19:03-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367205/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89f9a9935ad64a45a9954e48f9e5d827 Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)