2022-06-23T15:18:50.503 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-23T15:18:50.505 INFO:teuthology.results:subset: '1/999' 2022-06-23T15:18:50.505 INFO:teuthology.results:seed: '4579' 2022-06-23T15:31:26.740 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-23T15:31:26.743 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-23T16:06:29.845 INFO:teuthology.results:Tests finished! gathering results... 2022-06-23T16:06:31.201 INFO:scrape:Found 22 jobs 2022-06-23T16:06:33.301 INFO:scrape:Found 3 distinct failure reasons 2022-06-23T16:06:33.302 INFO:scrape:Failure: reached maximum tries (120) after waiting for 120 seconds 2022-06-23T16:06:33.303 INFO:scrape:6 jobs: ['6894700', '6894707', '6894696', '6894711', '6894704', '6894715'] 2022-06-23T16:06:33.303 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start'] 2022-06-23T16:06:33.304 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'rados:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'rados:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'rados:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'rados:cephadm/osds/{0-distro/ubuntu_20.04'] 2022-06-23T16:06:33.304 INFO:scrape: 2022-06-23T16:06:33.305 INFO:scrape:Failure: Command failed on smithi042 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe4c1424-f309-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi042:/dev/nvme4n1' 2022-06-23T16:06:33.305 INFO:scrape:7 jobs: ['6894708', '6894699', '6894695', '6894716', '6894702', '6894713', '6894705'] 2022-06-23T16:06:33.305 INFO:scrape:suites intersection: ['0-nvme-loop', 'fixed-2', 'start}'] 2022-06-23T16:06:33.306 INFO:scrape:suites union: ['0-nvme-loop', 'agent/off', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'rados:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'rados:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'rados:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'rados:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'rados:cephadm/smoke/{0-distro/ubuntu_20.04', 'start}'] 2022-06-23T16:06:33.306 INFO:scrape: 2022-06-23T16:06:33.307 INFO:scrape:Failure: timeout expired in wait_until_healthy 2022-06-23T16:06:33.307 INFO:scrape:3 jobs: ['6894703', '6894697', '6894709'] 2022-06-23T16:06:33.308 INFO:scrape:suites intersection: [] 2022-06-23T16:06:33.308 INFO:scrape:suites union: ['1-start', '2-services/basic', '2-services/rgw', '3-final}', 'mon_election/connectivity', 'rados:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools}', 'rados:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'rados:cephadm/workunits/{agent/off', 'task/test_nfs}'] 2022-06-23T16:06:33.309 INFO:scrape: 2022-06-23T16:06:33.310 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/ failed: 16 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [6894695] rados:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:40 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894695/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54fa1935a3e14653b565c5b01d23b949 Command failed on smithi157 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38be9dee-f309-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi157:/dev/nvme4n1' [6894696] rados:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:29 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894696/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58c167d7297c4c12a7a0b03f486862de reached maximum tries (120) after waiting for 120 seconds [6894697] rados:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:19:27 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894697/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d98ac6755b104229a5d77e26f7e12db2 timeout expired in wait_until_healthy [6894699] rados:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:20 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894699/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ceae36ca91cc4cc5b26ef9dd5052721c Command failed on smithi112 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2d75684a-f30a-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi112:/dev/nvme4n1' [6894700] rados:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:14:18 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894700/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894700/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1bd6f673ea6f4c349baff6d85c4a07e1 reached maximum tries (120) after waiting for 120 seconds [6894702] rados:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:51 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894702/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42f20bfa655848f69a4f84f0136e9840 Command failed on smithi074 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 522774f8-f30a-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi074:/dev/nvme4n1' [6894703] rados:cephadm/workunits/{agent/off mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:18:58 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894703/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894703/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d504b7bb847d4481813906aa23a99990 timeout expired in wait_until_healthy [6894704] rados:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:46 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894704/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a97b18bd3d6048d0ac570bef4d40de6e reached maximum tries (120) after waiting for 120 seconds [6894705] rados:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:36 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894705/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894705/ sentry: https://sentry.ceph.com/organizations/ceph/?query=967e438c05544243a3415dd1c0749f50 Command failed on smithi036 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 708e7108-f30a-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi036:/dev/nvme4n1' [6894707] rados:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:19:14 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894707/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894707/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e25eccde4654f778eb2825b716a7512 reached maximum tries (120) after waiting for 120 seconds [6894708] rados:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:39 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894708/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894708/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c184414e99564c33b17f35dc904f9563 Command failed on smithi042 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe4c1424-f309-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi042:/dev/nvme4n1' [6894709] rados:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:13:42 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894709/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894709/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8082623738224a6cb24c612efe1436b7 timeout expired in wait_until_healthy [6894711] rados:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:18:13 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894711/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894711/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc9c9fb1fe4e44139c6dc6a667bf7f3f reached maximum tries (120) after waiting for 120 seconds [6894713] rados:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:06:38 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894713/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894713/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05aa5f95d29e4bc49e80eaa9b9a41b38 Command failed on smithi107 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5a0c2600-f30a-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi107:/dev/nvme4n1' [6894715] rados:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:27:36 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894715/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa0757d4185040c48f16b81aea616da1 reached maximum tries (120) after waiting for 120 seconds [6894716] rados:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894716/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894716/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27a5c69b469d4ea5a8cb7513adc40397 Command failed on smithi160 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:df831b0959db84cbce1e3a370ed7a14414c1bf4a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70905fc6-f30b-11ec-842b-001a4aab830c -- ceph orch daemon add osd smithi160:/dev/nvme4n1' Pass ================================================================= [6894698] rados:cephadm/workunits/{agent/off mon_election/connectivity task/test_cephadm_repos} time: 00:04:17 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894698/ [6894701] rados:cephadm/workunits/{agent/on mon_election/classic task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}} time: 00:12:10 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894701/ [6894706] rados:cephadm/workunits/{agent/on mon_election/classic task/test_orch_cli} time: 00:17:12 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894706/ [6894710] rados:cephadm/workunits/{agent/on mon_election/classic task/test_adoption} time: 00:07:43 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894710/ [6894712] rados:cephadm/workunits/{agent/off mon_election/connectivity task/test_cephadm} time: 00:16:10 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894712/ [6894714] rados:cephadm/workunits/{agent/on mon_election/classic task/test_cephadm_repos} time: 00:04:20 info: http://pulpito.ceph.com/gabrioux-2022-06-23_15:14:26-rados:cephadm-wip-guits-testing-2022-06-23-1318-distro-default-smithi/6894714/