2023-07-26T05:24:09.928 INFO:root:teuthology version: 0.0.1.dev160+g407880c 2023-07-26T05:24:09.938 INFO:teuthology.results:subset: '1/8' 2023-07-26T05:24:09.939 INFO:teuthology.results:seed: '9062' 2023-07-26T06:05:21.802 INFO:root:teuthology version: 0.0.1.dev160+g407880c 2023-07-26T06:05:21.834 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-26T06:53:25.687 INFO:teuthology.results:Tests finished! gathering results... 2023-07-26T06:53:26.986 INFO:scrape:Found 22 jobs 2023-07-26T06:53:28.912 INFO:scrape:Found 3 distinct failure reasons 2023-07-26T06:53:28.912 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-07-26T06:53:28.913 INFO:scrape:2 jobs: ['7352774', '7352793'] 2023-07-26T06:53:28.913 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}'] 2023-07-26T06:53:28.913 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-07-26T06:53:28.914 INFO:scrape: 2023-07-26T06:53:28.914 INFO:scrape:Failure: Command failed on smithi137 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1029677a-2b79-11ee-9b35-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-07-26T06:53:28.914 INFO:scrape:4 jobs: ['7352782', '7352792', '7352779', '7352787'] 2023-07-26T06:53:28.914 INFO:scrape:suites intersection: ['4-wait', '5-upgrade-ls'] 2023-07-26T06:53:28.915 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-07-26T06:53:28.915 INFO:scrape: 2023-07-26T06:53:28.915 INFO:scrape:Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2023-07-26T06:53:28.916 INFO:scrape:1 jobs: ['7352780'] 2023-07-26T06:53:28.916 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_orch_cli}'] 2023-07-26T06:53:28.916 INFO:scrape: 2023-07-26T06:53:28.917 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [7352774] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:43:01 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352774/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a11098f08e8b437e9184018a88a46be2 reached maximum tries (301) after waiting for 300 seconds [7352779] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:45:05 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352779/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=add3784b39d041c09c9bf2950d4e7d3f Command failed on smithi079 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a60c15e-2b79-11ee-9b35-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7352780] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:45:20 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352780/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92e5e3e18c244599b6721879846c18ff Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7352782] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:35:20 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352782/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbc344d6de4d4cb18927979de9265523 Command failed on smithi137 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1029677a-2b79-11ee-9b35-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7352787] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:46:16 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352787/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da711f133add4e538e9841b8b0017d16 Command failed on smithi088 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f5b9994-2b7b-11ee-9b35-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7352792] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:35:22 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352792/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=659a65bf946b451aa8737b4d9870e685 Command failed on smithi122 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b97e6c42-2b7b-11ee-9b35-001a4aab830c -- bash -c 'ceph orch upgrade ls'" [7352793] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:28:41 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352793/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f49102142ce4789ad1f86f23289ea83 reached maximum tries (301) after waiting for 300 seconds Pass ================================================================= [7352773] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:21 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352773/ [7352775] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:41:50 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352775/ [7352776] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:47:27 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352776/ [7352777] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:39:45 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352777/ [7352778] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} time: 01:01:30 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352778/ [7352781] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 01:07:05 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352781/ [7352783] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:24:03 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352783/ [7352784] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:39:09 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352784/ [7352785] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:07:21 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352785/ [7352786] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:50:07 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352786/ [7352788] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} time: 00:18:55 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352788/ [7352789] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:46:22 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352789/ [7352790] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:36:33 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352790/ [7352791] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:25:25 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352791/ [7352794] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:21:02 info: http://pulpito.ceph.com/gabrioux-2023-07-26_05:21:54-orch:cephadm-wip-guits-testing-2-2023-07-25-0836-distro-default-smithi/7352794/