2024-02-12T23:43:17.848 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-12T23:43:17.849 INFO:teuthology.results:subset: '111/120000'
2024-02-12T23:43:17.849 INFO:teuthology.results:seed: '964'
2024-02-13T03:50:29.052 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-13T03:50:29.077 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-13T12:00:07.795 INFO:teuthology.results:Tests finished! gathering results...
2024-02-13T12:00:16.093 INFO:scrape:Found 104 jobs
2024-02-13T12:00:55.401 INFO:scrape:Found 16 distinct failure reasons
2024-02-13T12:00:55.402 INFO:scrape:Failure: "2024-02-13T00:08:10.908995+0000 mon.a (mon.0) 161 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log
2024-02-13T12:00:55.402 INFO:scrape:47 jobs
2024-02-13T12:00:55.403 INFO:scrape:suites intersection: []
2024-02-13T12:00:55.403 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-bootstrap/octopus', '1-install/mimic', '1-install/nautilus-v1only', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-verify}', '4-verify}}', '4-wait', '5-upgrade-ls', 'backoff/peering', 'backoff/peering_and_degraded', 'centos_8.stream_container_tools', 'ceph', 'clusters/{openstack', 'conf/{client', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'distro/ubuntu_18.04', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados', 'rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/osds/{0-distro/ubuntu_18.04', 'rados/cephadm/osds/{0-distro/ubuntu_20.04', 'rados/cephadm/smoke/{0-nvme-loop', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/with-work/{0-distro/ubuntu_18.04', 'rados/cephadm/with-work/{0-distro/ubuntu_20.04', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'roles', 'start', 'start}', 'task/test_e2e}', 'task/test_nfs}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/{0-from/v16.2.4', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/snaps-few-objects}']
2024-02-13T12:00:55.403 INFO:scrape:
2024-02-13T12:00:55.403 INFO:scrape:Failure: expected string or bytes-like object
2024-02-13T12:00:55.404 INFO:scrape:12 jobs: ['7557174', '7557201', '7557191', '7557133', '7557119', '7557167', '7557156', '7557181', '7557107', '7557125', '7557148', '7557142']
2024-02-13T12:00:55.404 INFO:scrape:suites intersection: ['1-start', '2-thrash', 'fixed-2', 'rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}']
2024-02-13T12:00:55.404 INFO:scrape:suites union: ['1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'fixed-2', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}']
2024-02-13T12:00:55.404 INFO:scrape:
2024-02-13T12:00:55.404 INFO:scrape:Failure: Failed to reconnect to smithi175
2024-02-13T12:00:55.404 INFO:scrape:3 jobs: ['7557158', '7557139', '7557126']
2024-02-13T12:00:55.404 INFO:scrape:suites intersection: []
2024-02-13T12:00:55.405 INFO:scrape:suites union: ['0-nvme-loop', '1-install/luminous-v1only', '1-install/mimic-v1only', '1-start', '2-services/nfs-ingress2', '3-final}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr-failures/few', 'rados', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/careful', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/snaps-few-objects}']
2024-02-13T12:00:55.405 INFO:scrape:
2024-02-13T12:00:55.405 INFO:scrape:Failure: "2024-02-13T03:40:00.000341+0000 mon.a (mon.0) 2253 : cluster [WRN]     use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." in cluster log
2024-02-13T12:00:55.405 INFO:scrape:3 jobs: ['7557182', '7557149', '7557121']
2024-02-13T12:00:55.405 INFO:scrape:suites intersection: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'start', 'tasks/rados_api_tests}']
2024-02-13T12:00:55.405 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'start', 'tasks/rados_api_tests}']
2024-02-13T12:00:55.405 INFO:scrape:
2024-02-13T12:00:55.405 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds
2024-02-13T12:00:55.406 INFO:scrape:4 jobs: ['7557194', '7557200', '7557197', '7557188']
2024-02-13T12:00:55.406 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}']
2024-02-13T12:00:55.406 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04']
2024-02-13T12:00:55.406 INFO:scrape:
2024-02-13T12:00:55.406 INFO:scrape:Failure: Command failed on smithi032 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2adf8f4-ca1e-11ee-95b9-87774f69a715 -e sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\''
2024-02-13T12:00:55.406 INFO:scrape:4 jobs: ['7557177', '7557199', '7557129', '7557120']
2024-02-13T12:00:55.406 INFO:scrape:suites intersection: []
2024-02-13T12:00:55.406 INFO:scrape:suites union: ['1-bootstrap/octopus', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9']
2024-02-13T12:00:55.406 INFO:scrape:
2024-02-13T12:00:55.407 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1
2024-02-13T12:00:55.407 INFO:scrape:2 jobs: ['7557189', '7557140']
2024-02-13T12:00:55.407 INFO:scrape:suites intersection: ['rados/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}']
2024-02-13T12:00:55.407 INFO:scrape:suites union: ['rados/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}']
2024-02-13T12:00:55.407 INFO:scrape:
2024-02-13T12:00:55.407 INFO:scrape:Failure: Command failed on smithi188 with status 5: 'sudo systemctl stop ceph-93a1e532-ca20-11ee-95b9-87774f69a715@mon.smithi188'
2024-02-13T12:00:55.407 INFO:scrape:6 jobs: ['7557190', '7557185', '7557176', '7557164', '7557198', '7557165']
2024-02-13T12:00:55.407 INFO:scrape:suites intersection: ['3-upgrade-with-workload']
2024-02-13T12:00:55.407 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-final}', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'roles', 'tasks/{0-from/v16.2.4']
2024-02-13T12:00:55.408 INFO:scrape:
2024-02-13T12:00:55.408 INFO:scrape:Failure: Command failed on smithi005 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 84e714279aa684636ac645daf0a1a85c2094cfb3'
2024-02-13T12:00:55.408 INFO:scrape:1 jobs: ['7557151']
2024-02-13T12:00:55.408 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'roles', 'tasks/{0-from/v16.2.4']
2024-02-13T12:00:55.408 INFO:scrape:
2024-02-13T12:00:55.408 INFO:scrape:Failure: Command failed on smithi032 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d15b559a-ca05-11ee-95b9-87774f69a715 -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\''
2024-02-13T12:00:55.408 INFO:scrape:1 jobs: ['7557144']
2024-02-13T12:00:55.408 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'rados/cephadm/osds/{0-distro/ubuntu_18.04']
2024-02-13T12:00:55.408 INFO:scrape:
2024-02-13T12:00:55.409 INFO:scrape:Failure: "2024-02-13T01:30:00.000184+0000 mon.a (mon.0) 2172 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 36484/263163 objects degraded (13.864%), 8 pgs degraded, 12 pgs undersized" in cluster log
2024-02-13T12:00:55.409 INFO:scrape:1 jobs: ['7557153']
2024-02-13T12:00:55.409 INFO:scrape:suites: ['1-install/nautilus-v2only', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}']
2024-02-13T12:00:55.409 INFO:scrape:
2024-02-13T12:00:55.409 INFO:scrape:Dead: b'2024-02-13T11:53:42.945 DEBUG:teuthology.exit:Finished running handlers'
2024-02-13T12:00:55.409 INFO:scrape:1 jobs: ['7557101']
2024-02-13T12:00:55.409 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{ubuntu_latest}}']
2024-02-13T12:00:55.409 INFO:scrape:
2024-02-13T12:00:55.409 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=84e714279aa684636ac645daf0a1a85c2094cfb3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2024-02-13T12:00:55.409 INFO:scrape:1 jobs: ['7557196']
2024-02-13T12:00:55.409 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}']
2024-02-13T12:00:55.410 INFO:scrape:
2024-02-13T12:00:55.410 INFO:scrape:Failure: "2024-02-13T00:30:09.964725+0000 mds.foofs.smithi003.hpgbbg (mds.0) 2 : cluster [WRN] client session with duplicated session uuid 'ganesha-nfs.foo.0-0001' denied (client.14712 172.21.15.3:0/1402965498)" in cluster log
2024-02-13T12:00:55.410 INFO:scrape:1 jobs: ['7557132']
2024-02-13T12:00:55.410 INFO:scrape:suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools']
2024-02-13T12:00:55.410 INFO:scrape:
2024-02-13T12:00:55.410 INFO:scrape:Failure: 'package_manager_version'
2024-02-13T12:00:55.410 INFO:scrape:2 jobs: ['7557166', '7557118']
2024-02-13T12:00:55.410 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'openstack}', 'pool/datapool', 'rados/cephadm/rbd_iscsi/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/ceph_iscsi}']
2024-02-13T12:00:55.410 INFO:scrape:suites union: ['cluster/{fixed-3', 'openstack}', 'pool/datapool', 'rados/cephadm/rbd_iscsi/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/ceph_iscsi}']
2024-02-13T12:00:55.410 INFO:scrape:
2024-02-13T12:00:55.410 INFO:scrape:Failure: Command failed on smithi028 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.28:/foouser /mnt/foo'"
2024-02-13T12:00:55.411 INFO:scrape:1 jobs: ['7557114']
2024-02-13T12:00:55.411 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04']
2024-02-13T12:00:55.411 INFO:scrape: