2023-05-22T14:03:52.798 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e 2023-05-22T14:03:52.822 INFO:teuthology.results:subset: '1/999' 2023-05-22T14:03:52.822 INFO:teuthology.results:seed: '5085' 2023-05-22T14:04:42.885 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e 2023-05-22T14:04:42.889 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-22T16:18:53.623 INFO:teuthology.results:Tests finished! gathering results... 2023-05-22T16:18:54.705 INFO:scrape:Found 15 jobs 2023-05-22T16:18:55.974 INFO:scrape:Found 6 distinct failure reasons 2023-05-22T16:18:55.975 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4bc7393ca73304444f3b8f1b3fc35efadb55ff4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-05-22T16:18:55.975 INFO:scrape:1 jobs: ['7282405'] 2023-05-22T16:18:55.976 INFO:scrape:suites: ['orch:cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2023-05-22T16:18:55.976 INFO:scrape: 2023-05-22T16:18:55.976 INFO:scrape:Failure: SSH connection to smithi007 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' 2023-05-22T16:18:55.976 INFO:scrape:1 jobs: ['7282418'] 2023-05-22T16:18:55.977 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04'] 2023-05-22T16:18:55.977 INFO:scrape: 2023-05-22T16:18:55.977 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-05-22T16:18:55.978 INFO:scrape:2 jobs: ['7282408', '7282415'] 2023-05-22T16:18:55.978 INFO:scrape:suites intersection: ['orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}'] 2023-05-22T16:18:55.978 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}'] 2023-05-22T16:18:55.978 INFO:scrape: 2023-05-22T16:18:55.979 INFO:scrape:Failure: Command failed on smithi007 with status 100: 'sudo apt-get clean' 2023-05-22T16:18:55.979 INFO:scrape:1 jobs: ['7282417'] 2023-05-22T16:18:55.979 INFO:scrape:suites: ['distro/ubuntu_18.04', 'fixed-2', 'mon_election/classic', 'orch:cephadm/smoke/{0-nvme-loop', 'start}'] 2023-05-22T16:18:55.980 INFO:scrape: 2023-05-22T16:18:55.980 INFO:scrape:Failure: Cannot connect to remote host smithi026 2023-05-22T16:18:55.980 INFO:scrape:1 jobs: ['7282414'] 2023-05-22T16:18:55.980 INFO:scrape:suites: ['1-install/nautilus-v2only', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/test_rbd_api}'] 2023-05-22T16:18:55.981 INFO:scrape: 2023-05-22T16:18:55.981 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f4bc7393ca73304444f3b8f1b3fc35efadb55ff4 2023-05-22T16:18:55.981 INFO:scrape:1 jobs: ['7282410'] 2023-05-22T16:18:55.982 INFO:scrape:suites: ['orch:cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2023-05-22T16:18:55.982 INFO:scrape: 2023-05-22T16:18:55.983 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/ failed: 6 dead: 1 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7282405] orch:cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:27:32 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282405/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7da1249e11448ae86003db059e6553d Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4bc7393ca73304444f3b8f1b3fc35efadb55ff4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7282408] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:24:41 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282408/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56ff049482ce4a50a6883e92f47825b8 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7282410] orch:cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:08:02 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282410/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d315b80c829e4651bc22a01661eb048b Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =f4bc7393ca73304444f3b8f1b3fc35efadb55ff4 [7282414] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282414/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2cadca00a5eb44b1b00ac6ef3ea14325 Cannot connect to remote host smithi026 [7282415] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:25:03 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282415/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5bd1e17619bd408a91e95af44bdedaea Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7282417] orch:cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282417/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282417/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13b99ba776b44d9d964788f24d801517 Command failed on smithi007 with status 100: 'sudo apt-get clean' Dead ================================================================= [7282418] orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282418/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2b102b90feb4e0fba9d309cf9b6b70a SSH connection to smithi007 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' Pass ================================================================= [7282406] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 02:01:21 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282406/ [7282407] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:28:52 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282407/ [7282409] orch:cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:22:10 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282409/ [7282411] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:42:28 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282411/ [7282412] orch:cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:18:38 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282412/ [7282413] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:54:52 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282413/ [7282416] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:50:41 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282416/ [7282419] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:56:43 info: http://pulpito.ceph.com/adking-2023-05-22_14:03:36-orch:cephadm-wip-adk4-testing-2023-05-21-1612-pacific-distro-default-smithi/7282419/