2023-07-13T15:28:05.225 INFO:root:teuthology version: 0.0.1.dev143+g073e6eb 2023-07-13T15:28:05.227 INFO:teuthology.results:subset: '1/999' 2023-07-13T15:28:05.227 INFO:teuthology.results:seed: '3772' 2023-07-13T15:52:51.641 INFO:root:teuthology version: 0.0.1.dev143+g073e6eb 2023-07-13T15:52:51.644 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-13T17:46:59.376 INFO:teuthology.results:Tests finished! gathering results... 2023-07-13T17:47:00.048 INFO:scrape:Found 17 jobs 2023-07-13T17:47:00.486 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336197/teuthology.log 2023-07-13T17:47:00.867 INFO:scrape:Found 7 distinct failure reasons 2023-07-13T17:47:00.868 INFO:scrape:Failure: 'package_manager_version' 2023-07-13T17:47:00.868 INFO:scrape:1 jobs: ['7336187'] 2023-07-13T17:47:00.868 INFO:scrape:suites: ['cluster/{fixed-3', 'openstack}', 'orch:cephadm/rbd_iscsi/{base/install', 'pool/datapool', 'supported-random-distro$/{centos_8}', 'workloads/ceph_iscsi}'] 2023-07-13T17:47:00.869 INFO:scrape: 2023-07-13T17:47:00.869 INFO:scrape:Failure: Cannot connect to remote host smithi007 2023-07-13T17:47:00.869 INFO:scrape:3 jobs: ['7336196', '7336188', '7336189'] 2023-07-13T17:47:00.869 INFO:scrape:suites intersection: ['ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados', 'thrashosds-health', 'three-plus-one}'] 2023-07-13T17:47:00.870 INFO:scrape:suites union: ['1-install/mimic-v1only', '1-install/nautilus-v1only', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados', 'thrashers/careful', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/snaps-few-objects}'] 2023-07-13T17:47:00.870 INFO:scrape: 2023-07-13T17:47:00.870 INFO:scrape:Failure: Command failed on smithi160 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fed8865a-219a-11ee-9b32-001a4aab830c -- ceph mon dump -f json' 2023-07-13T17:47:00.871 INFO:scrape:1 jobs: ['7336202'] 2023-07-13T17:47:00.871 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4'] 2023-07-13T17:47:00.871 INFO:scrape: 2023-07-13T17:47:00.872 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-07-13T17:47:00.872 INFO:scrape:2 jobs: ['7336198', '7336190'] 2023-07-13T17:47:00.872 INFO:scrape:suites intersection: ['orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}'] 2023-07-13T17:47:00.873 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}'] 2023-07-13T17:47:00.873 INFO:scrape: 2023-07-13T17:47:00.873 INFO:scrape:Dead: None 2023-07-13T17:47:00.873 INFO:scrape:1 jobs: ['7336197'] 2023-07-13T17:47:00.874 INFO:scrape: 2023-07-13T17:47:00.874 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=78d0ef1aa8678d5dee35caad62b7f72716369c35 2023-07-13T17:47:00.874 INFO:scrape:1 jobs: ['7336193'] 2023-07-13T17:47:00.874 INFO:scrape:suites: ['orch:cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2023-07-13T17:47:00.875 INFO:scrape: 2023-07-13T17:47:00.875 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78d0ef1aa8678d5dee35caad62b7f72716369c35 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-07-13T17:47:00.875 INFO:scrape:1 jobs: ['7336186'] 2023-07-13T17:47:00.876 INFO:scrape:suites: ['orch:cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2023-07-13T17:47:00.876 INFO:scrape: 2023-07-13T17:47:00.877 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/ failed: 9 dead: 1 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7336186] orch:cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:30:00 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336186/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0b92d3d0af94e95802d304287f3fc53 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78d0ef1aa8678d5dee35caad62b7f72716369c35 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7336187] orch:cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{centos_8} workloads/ceph_iscsi} ----------------------------------------------------------------- time: 00:17:03 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336187/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fe81024e1864b109eef3b21b403f32c 'package_manager_version' [7336188] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336188/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a4174e879154513bee41bdba21e62f9 Cannot connect to remote host smithi193 [7336189] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336189/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=824153cae8c44e0ea99d0cd0501315f4 Cannot connect to remote host smithi186 [7336190] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:28:22 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336190/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58e01513ffd2452288e383d6f8e7e804 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7336193] orch:cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:07:49 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336193/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2e97796744445f9b6ebdb02665ca867 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =78d0ef1aa8678d5dee35caad62b7f72716369c35 [7336196] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336196/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=157b6423e4de4cccbc528fd4b13ccc98 Cannot connect to remote host smithi007 [7336198] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:47:06 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336198/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336198/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de40b224811f4e7ca11170598860f007 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7336202] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:28:56 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336202/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=100c74a401754e0bb78d2cac43fa5be0 Command failed on smithi160 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fed8865a-219a-11ee-9b32-001a4aab830c -- ceph mon dump -f json' Dead ================================================================= [7336197] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336197/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336197/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds Pass ================================================================= [7336191] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:54:24 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336191/ [7336192] orch:cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:42:40 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336192/ [7336194] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:57:25 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336194/ [7336195] orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:20:52 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336195/ [7336199] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:31:36 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336199/ [7336200] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:37:57 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336200/ [7336201] orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:21:10 info: http://pulpito.ceph.com/adking-2023-07-13_13:35:25-orch:cephadm-wip-adk2-testing-2023-07-12-1752-pacific-distro-default-smithi/7336201/