2024-02-11T15:34:15.023 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-11T15:34:15.024 INFO:teuthology.results:subset: '111/120000'
2024-02-11T15:34:15.024 INFO:teuthology.results:seed: '1730'
2024-02-11T16:41:22.179 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-11T16:41:22.205 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-11T18:02:28.775 INFO:teuthology.results:Tests finished! gathering results...
2024-02-11T18:02:32.129 INFO:scrape:Found 49 jobs
2024-02-11T18:02:37.021 INFO:scrape:Found 8 distinct failure reasons
2024-02-11T18:02:37.022 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5db25abe00e218b6920215c0bef70af7e47c7152 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2024-02-11T18:02:37.022 INFO:scrape:1 jobs: ['7556243']
2024-02-11T18:02:37.022 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}']
2024-02-11T18:02:37.022 INFO:scrape:
2024-02-11T18:02:37.022 INFO:scrape:Failure: Failed to reconnect to smithi181
2024-02-11T18:02:37.022 INFO:scrape:3 jobs: ['7556218', '7556226', '7556242']
2024-02-11T18:02:37.022 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start']
2024-02-11T18:02:37.022 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '3-final}', 'rados/cephadm/osds/{0-distro/ubuntu_18.04', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04']
2024-02-11T18:02:37.022 INFO:scrape:
2024-02-11T18:02:37.023 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds
2024-02-11T18:02:37.023 INFO:scrape:3 jobs: ['7556246', '7556239', '7556244']
2024-02-11T18:02:37.023 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}']
2024-02-11T18:02:37.023 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress2', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04']
2024-02-11T18:02:37.023 INFO:scrape:
2024-02-11T18:02:37.023 INFO:scrape:Failure: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178'
2024-02-11T18:02:37.023 INFO:scrape:4 jobs: ['7556240', '7556245', '7556237', '7556238']
2024-02-11T18:02:37.023 INFO:scrape:suites intersection: ['3-upgrade-with-workload']
2024-02-11T18:02:37.023 INFO:scrape:suites union: ['1-bootstrap/16.2.5', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-final}', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'roles', 'tasks/{0-from/v16.2.4']
2024-02-11T18:02:37.023 INFO:scrape:
2024-02-11T18:02:37.024 INFO:scrape:Failure: 'package_manager_version'
2024-02-11T18:02:37.024 INFO:scrape:2 jobs: ['7556230', '7556205']
2024-02-11T18:02:37.024 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'openstack}', 'pool/datapool', 'rados/cephadm/rbd_iscsi/{base/install', 'workloads/ceph_iscsi}']
2024-02-11T18:02:37.024 INFO:scrape:suites union: ['cluster/{fixed-3', 'openstack}', 'pool/datapool', 'rados/cephadm/rbd_iscsi/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/ceph_iscsi}']
2024-02-11T18:02:37.024 INFO:scrape:
2024-02-11T18:02:37.024 INFO:scrape:Failure: timeout expired in wait_until_healthy
2024-02-11T18:02:37.024 INFO:scrape:2 jobs: ['7556235', '7556234']
2024-02-11T18:02:37.024 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'roles', 'tasks/{0-from/v16.2.4']
2024-02-11T18:02:37.024 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'roles', 'tasks/{0-from/v16.2.4']
2024-02-11T18:02:37.025 INFO:scrape:
2024-02-11T18:02:37.025 INFO:scrape:Failure: "2024-02-11T17:04:28.268688+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2024-02-11T18:02:37.025 INFO:scrape:2 jobs: ['7556241', '7556215']
2024-02-11T18:02:37.025 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/lockdep}']
2024-02-11T18:02:37.025 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/lockdep}']
2024-02-11T18:02:37.025 INFO:scrape:
2024-02-11T18:02:37.025 INFO:scrape:Failure: Command failed on smithi120 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.120:/foouser /mnt/foo'"
2024-02-11T18:02:37.025 INFO:scrape:1 jobs: ['7556202']
2024-02-11T18:02:37.025 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04']
2024-02-11T18:02:37.025 INFO:scrape: