2023-08-11T07:14:43.193 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-11T07:14:43.214 INFO:teuthology.results:subset: '1/8' 2023-08-11T07:14:43.215 INFO:teuthology.results:seed: '2550' 2023-08-11T07:21:51.129 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-11T07:21:51.132 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-11T08:22:55.994 INFO:teuthology.results:Tests finished! gathering results... 2023-08-11T08:22:56.906 INFO:scrape:Found 17 jobs 2023-08-11T08:23:02.010 INFO:scrape:Found 4 distinct failure reasons 2023-08-11T08:23:02.011 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-08-11T08:23:02.012 INFO:scrape:5 jobs: ['7367152', '7367150', '7367163', '7367165', '7367160'] 2023-08-11T08:23:02.012 INFO:scrape:suites intersection: ['1-start'] 2023-08-11T08:23:02.012 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/16.2.0', '1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-start', '2-nfs', '2-services/jaeger', '3-final}', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-08-11T08:23:02.013 INFO:scrape: 2023-08-11T08:23:02.013 INFO:scrape:Failure: Command failed on smithi081 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.81:/fake /mnt/foo'" 2023-08-11T08:23:02.013 INFO:scrape:3 jobs: ['7367155', '7367159', '7367158'] 2023-08-11T08:23:02.014 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-08-11T08:23:02.014 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-08-11T08:23:02.015 INFO:scrape: 2023-08-11T08:23:02.015 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2023-08-11T08:23:02.015 INFO:scrape:3 jobs: ['7367166', '7367161', '7367151'] 2023-08-11T08:23:02.016 INFO:scrape:suites intersection: ['task/test_nfs}'] 2023-08-11T08:23:02.016 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_nfs}'] 2023-08-11T08:23:02.016 INFO:scrape: 2023-08-11T08:23:02.016 INFO:scrape:Failure: Command failed on smithi093 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e06df16-381a-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-08-11T08:23:02.017 INFO:scrape:4 jobs: ['7367162', '7367153', '7367164', '7367157'] 2023-08-11T08:23:02.017 INFO:scrape:suites intersection: ['4-wait', '5-upgrade-ls'] 2023-08-11T08:23:02.017 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-08-11T08:23:02.018 INFO:scrape: 2023-08-11T08:23:02.019 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/ failed: 15 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7367150] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:40:34 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367150/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adab5fb8393a47b5a277e9cbcad431d3 reached maximum tries (301) after waiting for 300 seconds [7367151] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:33:56 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367151/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6142ac1e23246d2af36df27a86534d7 Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) [7367152] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:49:42 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367152/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3162342f33e84cea9bc8ce2d52aaf730 reached maximum tries (301) after waiting for 300 seconds [7367153] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:44:42 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367153/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58e837cebc7f4f8d97ee75f8b7d04305 Command failed on smithi131 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 34d5c3dc-3819-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367155] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:18:00 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367155/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367155/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01a0280d6dc44de48605d5d3023d3fe8 Command failed on smithi081 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.81:/fake /mnt/foo'" [7367157] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:36:52 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367157/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367157/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ae7c72878594ebf86c9893ce17b0727 Command failed on smithi060 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5c1ef51c-3819-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367158] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:24:02 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367158/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367158/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04596b8f7958473b804bb8da191bfe20 Command failed on smithi122 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.122:/fake /mnt/foo -o port=2999'" [7367159] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:21:01 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367159/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367159/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b62e460bd66d48048a6a9fff61a0b704 Command failed on smithi167 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.167:/fake /mnt/foo'" [7367160] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:50:58 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367160/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367160/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7be0931ca0e6452cbc50711e5bedb062 reached maximum tries (301) after waiting for 300 seconds [7367161] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:34:08 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367161/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367161/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8d1a3984e814e70a47a211a117f1b63 Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) [7367162] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:44:49 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367162/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367162/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d322b7b17bb54b49bb2bf3e2a59d8d87 Command failed on smithi093 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e06df16-381a-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367163] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:49:47 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367163/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367163/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffba2d1f24a94a0f852b0c06c4a16bcd reached maximum tries (301) after waiting for 300 seconds [7367164] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:35:52 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367164/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367164/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7d8808fe4404a72aac1fd1bef62e997 Command failed on smithi186 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid abb3c49a-3819-11ee-9b37-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7367165] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:28:41 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367165/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367165/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bee21bd71f9449dba03ebd782b2d6b12 reached maximum tries (301) after waiting for 300 seconds [7367166] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:26:04 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367166/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367166/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8fe8c0bf35934256a476fa238778d285 Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) Pass ================================================================= [7367154] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:23:28 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367154/ [7367156] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} time: 00:31:11 info: http://pulpito.ceph.com/gabrioux-2023-08-11_07:13:53-orch:cephadm-wip-guits-testing-2-2023-08-10-1325-distro-default-smithi/7367156/