2023-09-12T12:25:06.470 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-12T12:25:06.471 INFO:teuthology.results:subset: '1/14' 2023-09-12T12:25:06.471 INFO:teuthology.results:seed: '8048' 2023-09-12T12:47:45.671 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-12T12:47:45.674 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-12T13:23:48.345 INFO:teuthology.results:Tests finished! gathering results... 2023-09-12T13:23:48.863 INFO:scrape:Found 22 jobs 2023-09-12T13:23:49.791 INFO:scrape:Found 4 distinct failure reasons 2023-09-12T13:23:49.792 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-09-12T13:23:49.793 INFO:scrape:11 jobs: ['7394946', '7394950', '7394963', '7394954', '7394965', '7394953', '7394962', '7394949', '7394952', '7394966', '7394957'] 2023-09-12T13:23:49.793 INFO:scrape:suites intersection: ['fixed-2'] 2023-09-12T13:23:49.794 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'root}', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}'] 2023-09-12T13:23:49.794 INFO:scrape: 2023-09-12T13:23:49.795 INFO:scrape:Failure: Command failed on smithi064 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid baa3d2e2-5169-11ee-9ab7-7b867c8bd7da -e sha1=88c016b40bb78c1bdad993249cf9faa92f6877be -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-09-12T13:23:49.795 INFO:scrape:1 jobs: ['7394947'] 2023-09-12T13:23:49.795 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-09-12T13:23:49.796 INFO:scrape: 2023-09-12T13:23:49.796 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-09-12T13:23:49.796 INFO:scrape:1 jobs: ['7394958'] 2023-09-12T13:23:49.797 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-09-12T13:23:49.797 INFO:scrape: 2023-09-12T13:23:49.798 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-09-12T13:23:49.798 INFO:scrape:1 jobs: ['7394948'] 2023-09-12T13:23:49.799 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cluster/{1-node}', 'objectstore/bluestore-bitmap', 'orch:cephadm/nfs/{begin/{0-install', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_8}', 'tasks/nfs}'] 2023-09-12T13:23:49.799 INFO:scrape: 2023-09-12T13:23:49.800 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/ failed: 14 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7394946] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:23:59 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394946/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394946/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c0c50a04eff42ff9839eb793e94ca5f timeout expired in wait_until_healthy [7394947] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:31:42 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394947/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394947/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2267cd2e273f4667b949cb9827a25892 Command failed on smithi064 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid baa3d2e2-5169-11ee-9ab7-7b867c8bd7da -e sha1=88c016b40bb78c1bdad993249cf9faa92f6877be -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7394948] orch:cephadm/nfs/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/nfs} ----------------------------------------------------------------- time: 00:17:06 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394948/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef83ad1752514f95b5c146418ccd5835 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7394949] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:24:06 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394949/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394949/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f3025e2b5da436c90a4bf2c68e53b31 timeout expired in wait_until_healthy [7394950] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:22:53 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394950/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394950/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a787f6996984bd8927fb78fdaca91c8 timeout expired in wait_until_healthy [7394952] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:30:03 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394952/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58518d74070b4b43828dd702353caeb8 timeout expired in wait_until_healthy [7394953] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:29:28 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394953/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66660093d5c940c3bc847cf539d9f3c4 timeout expired in wait_until_healthy [7394954] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:28:10 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394954/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aff80df69afd469499eda3e95046e38e timeout expired in wait_until_healthy [7394957] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:28:37 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394957/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394957/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5ef123ae71c49bcb90e7b4e011801f1 timeout expired in wait_until_healthy [7394958] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:28:18 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394958/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bfb63770de14af58c308821b0b8bfe1 reached maximum tries (301) after waiting for 300 seconds [7394962] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:25:18 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394962/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394962/ sentry: https://sentry.ceph.com/organizations/ceph/?query=039ca4c04d3047b0a4e934bcabe1abfe timeout expired in wait_until_healthy [7394963] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:26:36 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394963/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=546596b76af2475b840209434465c689 timeout expired in wait_until_healthy [7394965] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:28:00 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394965/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394965/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26c9bebc868e4a1faa01e0aa25867f5a timeout expired in wait_until_healthy [7394966] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:22:52 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394966/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394966/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c398db78f7dd4ec78f9ccb60e561bc82 timeout expired in wait_until_healthy Pass ================================================================= [7394951] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:45:38 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394951/ [7394955] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:30:41 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394955/ [7394956] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:18:54 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394956/ [7394959] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:20:05 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394959/ [7394960] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:36:22 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394960/ [7394961] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:06 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394961/ [7394964] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:12:23 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394964/ [7394967] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:18:05 info: http://pulpito.ceph.com/adking-2023-09-12_12:23:05-orch:cephadm-wip-adk-testing-2023-09-11-1829-distro-default-smithi/7394967/