2023-07-11T10:48:16.879 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-11T10:48:16.880 INFO:teuthology.results:subset: '1/8' 2023-07-11T10:48:16.880 INFO:teuthology.results:seed: '5540' 2023-07-11T10:48:32.126 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-11T10:48:32.127 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-11T11:22:34.295 INFO:teuthology.results:Tests finished! gathering results... 2023-07-11T11:22:34.659 INFO:scrape:Found 16 jobs 2023-07-11T11:22:34.895 INFO:scrape:Found 2 distinct failure reasons 2023-07-11T11:22:34.896 INFO:scrape:Failure: Command failed on smithi037 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52bbb90c-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi037:vg_nvme/lv_4' 2023-07-11T11:22:34.896 INFO:scrape:12 jobs: ['7333631', '7333633', '7333634', '7333622', '7333629', '7333623', '7333626', '7333635', '7333624', '7333630', '7333628', '7333627'] 2023-07-11T11:22:34.897 INFO:scrape:suites intersection: [] 2023-07-11T11:22:34.897 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-thrash', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'start}', 'task/test_nfs}', 'task/test_rgw_multisite}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'workloads/cephadm_iscsi}'] 2023-07-11T11:22:34.897 INFO:scrape: 2023-07-11T11:22:34.898 INFO:scrape:Failure: reached maximum tries (121) after waiting for 120 seconds 2023-07-11T11:22:34.898 INFO:scrape:3 jobs: ['7333621', '7333636', '7333625'] 2023-07-11T11:22:34.898 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start'] 2023-07-11T11:22:34.898 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun'] 2023-07-11T11:22:34.899 INFO:scrape: 2023-07-11T11:22:34.900 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/ failed: 15 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7333621] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:17:42 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333621/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77665d32c9d1406da10a0811f85bf7dd reached maximum tries (121) after waiting for 120 seconds [7333622] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333622/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66dd94dff3ba453599fd70bf7cded86f Command failed on smithi072 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 234a2df2-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi072:vg_nvme/lv_4' [7333623] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:12:34 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333623/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d5d7bac6c2144ac8e1428902e8289fc Command failed on smithi090 with status 22: 'sudo cephadm --image quay- quay-quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ade5f6a-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi090:vg_nvme/lv_4' [7333624] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:20:59 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333624/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75d3929458bd47c1acdc9652a9c63c96 Command failed on smithi033 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a42af6c6-1fdc-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi033:vg_nvme/lv_4' [7333625] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:17:34 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333625/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333625/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1f95341092243f786140a5debcc8741 reached maximum tries (121) after waiting for 120 seconds [7333626] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333626/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333626/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab6cbcecbe2843a18f70e7049af18e26 Command failed on smithi060 with status 22: 'sudo cephadm --image quay- quay-quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ccc1bc6a-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi060:vg_nvme/lv_4' [7333627] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:10:27 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333627/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ba354eca21a4ce2a66cc04518fe58a2 Command failed on smithi026 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 857cfcb2-1fda-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi026:/dev/nvme4n1' [7333628] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:14:05 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333628/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57627ec4f8384905a3d06e0ac63629fe Command failed on smithi123 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6041104-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi123:vg_nvme/lv_4' [7333629] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:14:12 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333629/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b292319cbfb44a8586bc90080db6d8a6 Command failed on smithi027 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9a22bbba-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi027:vg_nvme/lv_4' [7333630] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:18:17 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333630/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333630/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fc9b4cd78c04dcda8fadb0ea44c92b2 Command failed on smithi097 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f3c48a-1fdc-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi097:vg_nvme/lv_4' [7333631] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:12:58 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333631/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333631/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19ab23d5af924190a44eaf9c692a5156 Command failed on smithi037 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52bbb90c-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi037:vg_nvme/lv_4' [7333633] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:19:24 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333633/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333633/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4e8e7027f014422a5f5b1becf4ec4b6 Command failed on smithi137 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a6ad7de-1fdc-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi137:vg_nvme/lv_4' [7333634] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:19 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333634/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333634/ sentry: https://sentry.ceph.com/organizations/ceph/?query=97de0b1d3e3140e5a60b430fc3baa61e Command failed on smithi104 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bbc88854-1fda-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi104:/dev/nvme4n1' [7333635] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:12:22 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333635/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333635/ sentry: https://sentry.ceph.com/organizations/ceph/?query=84e50e0d85b140c0b392f7df58d74f42 Command failed on smithi043 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:67d2dc08814f61c4316ce1f5c688a1abdd39d7c3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 05dc5704-1fdb-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi043:vg_nvme/lv_4' [7333636] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:17:11 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333636/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333636/ sentry: https://sentry.ceph.com/organizations/ceph/?query=066744ea51b54de093bac5f55369552b reached maximum tries (121) after waiting for 120 seconds Pass ================================================================= [7333632] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} time: 00:14:10 info: http://pulpito.ceph.com/gabrioux-2023-07-11_10:47:59-orch:cephadm-guits-cv-refactor-osd-objectstore-distro-default-smithi/7333632/