Found 323 jobs
Missing teuthology log /home/teuthworker/archive/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285910/teuthology.log
Found 15 distinct failure reasons
Failure: {'smithi107.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['rpm', '-e', 'ceph-debuginfo'], 'delta': '0:00:00.012461', 'end': '2023-05-25 02:11:42.006067', 'invocation': {'module_args': {'_raw_params': 'rpm -e ceph-debuginfo', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2023-05-25 02:11:41.993606', 'stderr': 'error: Failed dependencies:\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mds-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-cephfs-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-radosgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) libcephfs2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-mirror-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-base-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-test-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-osd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-nbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-common-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mgr-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librbd1-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados-devel-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mon-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rados-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', 'stderr_lines': ['error: Failed dependencies:', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mds-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-cephfs-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-radosgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) libcephfs2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-mirror-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-base-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-test-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-osd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-nbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-common-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mgr-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librbd1-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados-devel-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mon-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rados-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64'], 'stdout': '', 'stdout_lines': []}, 'smithi159.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': "  /dev/vg_nvme: already exists in filesystem\n  Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}}
1 jobs: ['7285901']
suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools']

Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2 jobs: ['7285672', '7285835']
suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}']
suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}']

Dead: None
11 jobs: ['7285910', '7285952', '7285853', '7285897', '7285961', '7285900', '7285894', '7285933', '7285946', '7285896', '7285943']

Failure: machine smithi038.front.sepia.ceph.com is locked by scheduled_teuthology@teuthology, not scheduled_yuriw@teuthology
1 jobs: ['7285859']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}']

Failure: Command failed (workunit test cls/test_cls_refcount.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh'
1 jobs: ['7285758']
suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']

Failure: Command failed on smithi079 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"
4 jobs: ['7285673', '7285755', '7285836', '7285916']
suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04']
suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}']

Failure: {'smithi100.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': "  /dev/vg_nvme: already exists in filesystem\n  Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}}
7 jobs: ['7285899', '7285903', '7285898', '7285902', '7285947', '7285948', '7285895']
suites intersection: []
suites union: ['agent/off', 'clusters/{fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'openstack', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/objectstore/{backends/fusestore', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/divergent_priors2', 'scheduler/wpq_default_shards', 'settings/optimized', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'task/test_nfs}', 'tasks/rados_workunit_loadgen_mostlyread}', 'tasks/repair_test}', 'ubuntu_latest', 'workloads/sample_fio}']

Failure: Cannot connect to remote host smithi177
2 jobs: ['7285958', '7285956']
suites intersection: []
suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', 'cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}']

Failure: Command failed on smithi023 with status 1: 'sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh collectl'
5 jobs: ['7285945', '7285944', '7285907', '7285949', '7285955']
suites intersection: []
suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{default}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'openstack', 'openstack}', 'rados', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'scheduler/dmclock_default_shards', 'settings/optimized', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashosds-health', 'ubuntu_latest', 'workloads/dedup-io-snaps}', 'workloads/ec-small-objects-overwrites}', 'workloads/fio_4K_rand_read}', 'workloads/redirect}']

Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS)
2 jobs: ['7285669', '7285807']
suites intersection: ['agent/off', 'mon_election/classic', 'task/test_nfs}']
suites union: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_nfs}']

Failure: {'smithi177.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi177.front.sepia.ceph.com port 22: No route to host'}}
1 jobs: ['7285909']
suites: ['0-nvme-loop', 'agent/off', 'fixed-2', 'mon_election/classic', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'start}']

Failure: Stale jobs detected, aborting.
1 jobs: ['7285918']
suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']

Failure: "2023-05-25T00:18:25.508837+0000 mon.a (mon.0) 2108 : cluster [WRN] Health check failed: 1 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log
1 jobs: ['7285745']
suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rhel_8}}']

Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh'
1 jobs: ['7285819']
suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{rhel_8}}']

Failure: SSH connection to smithi162 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'
1 jobs: ['7285908']
suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/version-number-sanity', 'supported-random-distro$/{rhel_8}}']