2023-09-19T20:02:04.286 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-19T20:02:04.287 INFO:teuthology.results:subset: '1/14' 2023-09-19T20:02:04.287 INFO:teuthology.results:seed: '9144' 2023-09-19T20:02:12.071 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-19T20:02:12.073 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-19T21:00:16.408 INFO:teuthology.results:Tests finished! gathering results... 2023-09-19T21:00:16.928 INFO:scrape:Found 11 jobs 2023-09-19T21:00:18.121 INFO:scrape:Found 3 distinct failure reasons 2023-09-19T21:00:18.122 INFO:scrape:Failure: Command failed on smithi121 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 395df0e8-572a-11ee-9ab8-7b867c8bd7da -e sha1=f7ec875ca4b13739f525f49d2ca0ee200065c5f0 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' 2023-09-19T21:00:18.123 INFO:scrape:2 jobs: ['7398343', '7398336'] 2023-09-19T21:00:18.123 INFO:scrape:suites intersection: [] 2023-09-19T21:00:18.124 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '3-upgrade-with-workload', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools'] 2023-09-19T21:00:18.124 INFO:scrape: 2023-09-19T21:00:18.124 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-09-19T21:00:18.125 INFO:scrape:1 jobs: ['7398337'] 2023-09-19T21:00:18.125 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cluster/{1-node}', 'objectstore/bluestore-bitmap', 'orch:cephadm/nfs/{begin/{0-install', 'overrides/ignorelist_health', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}'] 2023-09-19T21:00:18.126 INFO:scrape: 2023-09-19T21:00:18.126 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-09-19T21:00:18.127 INFO:scrape:1 jobs: ['7398344'] 2023-09-19T21:00:18.127 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-09-19T21:00:18.127 INFO:scrape: 2023-09-19T21:00:18.128 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7398336] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:35:19 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398336/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=081e049ed20b4a2c9b51d1717f8672ed Command failed on smithi145 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4ed0bcbc-572a-11ee-9ab8-7b867c8bd7da -e sha1=f7ec875ca4b13739f525f49d2ca0ee200065c5f0 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7398337] orch:cephadm/nfs/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:18:07 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398337/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b7cf876447e47b891e2b9d94657b48e Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7398343] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:41:59 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398343/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=634507ec2cf043b78f1b955454248137 Command failed on smithi121 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 395df0e8-572a-11ee-9ab8-7b867c8bd7da -e sha1=f7ec875ca4b13739f525f49d2ca0ee200065c5f0 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' [7398344] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:33:47 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398344/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80d304ae1cf44bc4ab99924626cb1640 reached maximum tries (301) after waiting for 300 seconds Pass ================================================================= [7398335] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:30:29 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398335/ [7398338] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:16:35 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398338/ [7398339] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:47:07 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398339/ [7398340] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:26:33 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398340/ [7398341] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:15:49 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398341/ [7398342] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:45:03 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398342/ [7398345] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:16:39 info: http://pulpito.ceph.com/adking-2023-09-19_20:01:57-orch:cephadm-wip-adk-testing-2023-09-18-1336-distro-default-smithi/7398345/